Clusters in Very LargeDatasets with High Dimensions

Multimedia databases typically contain data with very high dimensions. Finding interesting patterns in these databases poses a very challenging problem because of the scalability, lack of domain knowledge and complex structures of the embedded clusters. High dimensionality adds severely to the scalability problem. It has been shown that the wavelet-based clustering technique, WaveCluster, is very eecient and eeective in detecting arbitrary shape clusters and eliminating noisy data for low dimensional data. In this paper, we introduce HiperWave, an approach to applying wavelet-based techniques for clustering high dimensional data. Using a hash-based data structure, our approach makes intelligent use of available resources to discover clusters in the dataset. We demonstrate that the cost of clustering can be reduced dramatically yet maintaining all the advantages of wavelet-based clustering. This hash-based data representation can be applied for any grid-based clustering approaches. Finally, we introduce a quantitative metric to measure the quality of the resulting clusters. The experimental results show both eeectiveness and eeciency of our method on high dimensional datasets.